Automatic identification of workloads contributing to system performance degradation using machine learning techniques

ABSTRACT

Methods, apparatus, and processor-readable storage media for automatic identification of workloads contributing to system performance degradation are provided herein. An example computer-implemented method includes obtaining, in connection with a system exhibiting performance degradation, a primary time series and a set of multiple candidate time series; calculating, using machine learning, similarity measurements between the primary time series and each time series in the set; for each measurement, assigning weights to the time series based on similarity to the primary time series relative to the other time series in the set; generating, for each time series in the set, a similarity score based on the weights assigned across the similarity measurements; and outputting, based on the similarity scores, identification of a candidate time series for use in automated actions.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is related to U.S. patent application Ser. No.16/659,844, entitled “Determining Similarity Between Time Series UsingMachine Learning Techniques,” U.S. patent application Ser. No.16/659,862, entitled “Automatic Identification of Resources inContention in Storage Systems Using Machine Learning Techniques,” andU.S. patent application Ser. No. 16/659,872, entitled “AutomaticIdentification of Workloads Contributing to Behavioral Changes inStorage Systems Using Machine Learning Techniques,” each of which isfiled concurrently herewith and incorporated by reference herein.

FIELD

The field relates generally to information processing systems, and moreparticularly to techniques for processing time series data in suchsystems.

BACKGROUND

With dynamic systems, the behavior of various objects will typicallyvary over time. Additionally, many such objects are measured on atime-based regularity with respect to one or more variables. A sequenceof these measurements results in an object referred to as a time series,and the similarity between two time series within dynamic systems canprovide information in terms of the behavior of the measured entities.However, conventional time series analysis approaches face accuracy andscalability challenges.

SUMMARY

Illustrative embodiments of the disclosure provide methods for automaticidentification of workloads contributing to system performancedegradation using machine learning techniques. An exemplarycomputer-implemented method includes obtaining, in connection with asystem exhibiting performance degradation, a primary time series and aset of multiple candidate time series, wherein the primary time seriesincludes latency data attributed to the system or a targeted workloadwithin the system, and wherein the candidate time series includeinput/output activity data attributed to multiple workloads within thesystem. Such a method also includes calculating, using one or moremachine learning techniques, multiple similarity measurements betweenthe primary time series and each of the candidate time series in theset. Additionally, for each of the multiple similarity measurements,such a method includes assigning weights to the candidate time seriesbased at least in part on similarity to the primary time series relativeto the other candidate time series in the set. Further, such a methodalso includes generating, for each of the candidate time series in theset, a similarity score based at least in part on the weights assignedto each of the candidate time series across the multiple similaritymeasurements, and outputting, based at least in part on the similarityscores, identification of at least one candidate time series from theset for use in one or more automated actions associated with the systemin response to the performance degradation.

Illustrative embodiments can provide significant advantages relative toconventional time series analysis approaches. For example, challengesassociated with accuracy and scalability are overcome in one or moreembodiments through implementing a machine learning algorithm or othertypes of machine learning techniques that incorporate multiple distinctmeasures of similarity and distance between pairs of time series datapoints corresponding to latency and input/output activity.

These and other illustrative embodiments described herein include,without limitation, methods, apparatus, systems, and computer programproducts comprising processor-readable storage media.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an information processing system configured for determiningsimilarity between time series using machine learning techniques in anillustrative embodiment.

FIG. 2 shows a workflow for determining similarity between time seriesusing machine learning techniques in an illustrative embodiment.

FIG. 3 shows visual examples of similarity measures utilized in anillustrative embodiment.

FIG. 4 shows example pseudocode for a time series similarity algorithmin an illustrative embodiment.

FIG. 5 shows a workflow for automatic identification of workloadscontributing to system performance degradation using machine learningtechniques in an illustrative embodiment.

FIG. 6 is a flow diagram of a process for automatic identification ofworkloads contributing to system performance degradation using machinelearning techniques in an illustrative embodiment.

FIGS. 7 and 8 show examples of processing platforms that may be utilizedto implement at least a portion of an information processing system inillustrative embodiments.

DETAILED DESCRIPTION

Illustrative embodiments will be described herein with reference toexemplary information processing systems and associated computers,servers, storage devices and other processing devices. It is to beappreciated, however, that these and other embodiments are notrestricted to the particular illustrative system and deviceconfigurations shown. Accordingly, the term “information processingsystem” as used herein is intended to be broadly construed, so as toencompass, for example, processing systems comprising cloud computingand storage systems, as well as other types of processing systemscomprising various combinations of physical and virtual processingresources. An information processing system may therefore comprise, forexample, at least one data center or other cloud-based system thatincludes one or more clouds hosting multiple tenants that share cloudresources. Numerous different types of enterprise computing and storagesystems are also encompassed by the term “information processing system”as that term is broadly used herein.

FIG. 1 shows an information processing system 100 configured inaccordance with an illustrative embodiment. The information processingsystem 100 comprises a plurality of host devices 102-1, 102-2, . . .102-M, collectively referred to herein as host devices 102. The hostdevices 102 are coupled to a network 104. Also coupled to network 104 isstorage system 103. The host devices 102 are configured to communicatewith the storage system 103 over network 104.

The host devices 102 illustratively comprise servers or other types ofcomputers of an enterprise computer system, cloud-based computer systemor other arrangement of multiple compute nodes associated withrespective users. For example, the host devices 102 in some embodimentsillustratively provide compute services such as execution of one or moreapplications on behalf of each of one or more users associated withrespective ones of the host devices. In one or more embodiments, thehost devices 102 comprises a processor coupled to a memory. The hostdevices 102 are therefore an example of what is more generally referredto herein as a processing device comprising a processor coupled to amemory. The processor executes application processes of one or moreapplications on behalf of each of one or more users of the host device102. Such application process execution results in the generation ofwrite operations and read operations that are directed by the hostdevice 102 to the storage system 103 in the manner disclosed herein.

The storage system 103 illustratively comprises processing devices ofone or more processing platforms. For example, the storage system 103can comprise one or more processing devices each having a processor anda memory, possibly implementing virtual machines and/or containers,although numerous other configurations are possible.

The storage system 103 can additionally or alternatively be part ofcloud infrastructure such as an Amazon Web Services (AWS) system. Otherexamples of cloud-based systems that can be used to provide at leastportions of the storage system 103 include Google Cloud Platform (GCP)and Microsoft Azure.

The host devices 102 and the storage system 103 may be implemented on acommon processing platform, or on separate processing platforms. Thehost devices 102 are illustratively configured to write data to and readdata from the storage system 103 in accordance with applicationsexecuting on those host devices for system users.

Also, it is to be appreciated that the term “user” in this context andelsewhere herein is intended to be broadly construed so as to encompass,for example, human, hardware, software or firmware entities, as well asvarious combinations of such entities. Compute and/or storage servicesmay be provided for users under a Platform-as-a-Service (PaaS) model, anInfrastructure-as-a-Service (IaaS) model and/or a Function-as-a-Service(FaaS) model, although it is to be appreciated that numerous other cloudinfrastructure arrangements could be used. Also, illustrativeembodiments can be implemented outside of the cloud infrastructurecontext, as in the case of a stand-alone computing and storage systemimplemented within a given enterprise.

The network 104 is assumed to comprise a portion of a global computernetwork such as the Internet, although other types of networks can bepart of the network 104, including a wide area network (WAN), a localarea network (LAN), a satellite network, a telephone or cable network, acellular network, a wireless network such as a Wi-Fi or WiMAX network,or various portions or combinations of these and other types ofnetworks. The network 104 in some embodiments therefore comprisescombinations of multiple different types of networks, each comprisingprocessing devices configured to communicate using internet protocol(IP) or other related communication protocols.

As a more particular example, some embodiments may utilize one or morehigh-speed local networks in which associated processing devicescommunicate with one another utilizing Peripheral Component Interconnectexpress (PCIe) cards of those devices, and networking protocols such asInfiniBand, Gigabit Ethernet or Fibre Channel. Numerous alternativenetworking arrangements are possible in a given embodiment, as will beappreciated by those skilled in the art.

The storage system 103 comprises a plurality of storage devices 106 andan associated storage controller 108. The storage devices 106 store dataof a plurality of storage volumes 110. The storage volumes 110illustratively comprise respective logical units (LUNs) or other typesof logical storage volumes. It should be appreciated, however, that theterm “storage volume” as used herein is intended to be broadlyconstrued, and should not be viewed as being limited to any particularformat or configuration.

The storage devices 106 of the storage system 103 illustrativelycomprise solid state drives (SSDs). Such SSDs are implemented usingnon-volatile memory (NVM) devices such as flash memory. Other types ofNVM devices that can be used to implement at least a portion of thestorage devices 106 include non-volatile random access memory (NVRAM),phase-change RAM (PC-RAM), magnetic RAM (MRAM), resistive RAM, spintorque transfer magneto-resistive RAM (STT-MRAM), and Intel Optane™devices based on 3D XPoint™ memory. These and various combinations ofmultiple different types of NVM devices may also be used. For example,hard disk drives (HDDs) can be used in combination with or in place ofSSDs or other types of NVM devices.

However, it is to be appreciated that other types of storage devices canbe used in storage system 103 in other embodiments. For example, a givenstorage system, as the term is broadly used herein, can include acombination of different types of storage devices, as in the case of amulti-tier storage system comprising a flash-based fast tier and adisk-based capacity tier. In such an embodiment, each of the fast tierand the capacity tier of the multi-tier storage system comprises aplurality of storage devices with different types of storage devicesbeing used in different ones of the storage tiers. For example, the fasttier may comprise flash drives while the capacity tier comprises harddisk drives. The particular storage devices used in a given storage tiermay be varied in other embodiments, and multiple distinct storage devicetypes may be used within a single storage tier. The term “storagedevice” as used herein is intended to be broadly construed, so as toencompass, for example, SSDs, HDDs, flash drives, hybrid drives or othertypes of storage devices.

In some embodiments, the storage system 103 illustratively comprises ascale-out all-flash content addressable storage array such as anXtremIO™ storage array from Dell EMC of Hopkinton, Mass. A wide varietyof other types of storage arrays can be used in implementing a given oneof the storage system 103 in other embodiments, including by way ofexample one or more VNX®, VMAX®, Unity™ or PowerMax™ storage arrays,commercially available from Dell EMC. Additional or alternative types ofstorage products that can be used in implementing a given storage systemin illustrative embodiments include software-defined storage, cloudstorage, object-based storage and scale-out storage. Combinations ofmultiple ones of these and other storage types can also be used inimplementing a given storage system in an illustrative embodiment.

The term “storage system” as used herein is therefore intended to bebroadly construed, and should not be viewed as being limited to storagesystems based on flash memory or other types of NVM storage devices. Agiven storage system as the term is broadly used herein can comprise,for example, network-attached storage (NAS), storage area networks(SANs), direct-attached storage (DAS) and distributed DAS, as well ascombinations of these and other storage types, includingsoftware-defined storage.

In some embodiments, communications between the host devices 102 and thestorage system 103 comprise Small Computer System Interface (SCSI) orInternet SCSI (iSCSI) commands. Other types of SCSI or non-SCSI commandsmay be used in other embodiments, including commands that are part of astandard command set, or custom commands such as a “vendor uniquecommand” or VU command that is not part of a standard command set. Theterm “command” as used herein is therefore intended to be broadlyconstrued, so as to encompass, for example, a composite command thatcomprises a combination of multiple individual commands. Numerous othercommands can be used in other embodiments.

For example, although in some embodiments certain commands used by thehost devices 102 to communicate with the storage system 103illustratively comprise SCSI or iSCSI commands, other embodiments canimplement 10 operations utilizing command features and functionalityassociated with NVM Express (NVMe), as described in the NVMeSpecification, Revision 1.3, May 2017, which is incorporated byreference herein. Other storage protocols of this type that may beutilized in illustrative embodiments disclosed herein include NVMe overFabric, also referred to as NVMeoF, and NVMe over Transmission ControlProtocol (TCP), also referred to as NVMe/TCP.

The storage controller 108 of storage system 103 in the FIG. 1embodiment includes machine learning time series similarity algorithm105. The storage controller 108 can also include additional elements,such as replication control logic for controlling replication of one ormore of the storage volumes 110 to another storage system not shown inthe figure. The storage controller 108 and the storage system 103 mayfurther include one or more additional modules and other componentstypically found in conventional implementations of storage controllersand storage systems, although such additional modules and othercomponents are omitted from the figure for clarity and simplicity ofillustration.

It should be noted that one or more functionalities of storage system103 (including that provided the machine learning time series similarityalgorithm 105) as described herein with reference to host devices 102can additionally or alternatively be implemented by each of one or moreof the additional host devices 102. For example, each of the hostdevices 102 can be configured to include the machine learning timeseries similarity algorithm 105 of storage system 103.

The storage system 103 is illustratively implemented as a distributedstorage system, also referred to herein as a clustered storage system,in which such a storage system comprises a plurality of storage nodeseach comprising a set of processing modules configured to communicatewith corresponding sets of processing modules on other ones of thestorage nodes. The sets of processing modules of the storage nodes ofthe storage system 103 collectively comprise at least a portion of thestorage controller 108 of the storage system 103. For example, in someembodiments the sets of processing modules of the storage nodescollectively comprise a distributed storage controller of thedistributed storage system.

It is to be understood that the particular set of elements shown in FIG.1 for determining similarity between time series using machine learningtechniques involving host devices 102 is presented by way ofillustrative example only, and in other embodiments additional oralternative elements may be used. Thus, another embodiment includesadditional or alternative systems, devices and other network entities,as well as different arrangements of modules and other components.

An exemplary process utilizing an example machine learning time seriessimilarity algorithm 105 will be described in more detail with referenceto the flow diagram of FIG. 6.

Accordingly, at least one embodiment of the invention includesgenerating and/or implementing a machine learning algorithm formeasuring similarity between a baseline/primary time series and a set ofother time series to determine a subset of the other time series thatmost closely resembles the baseline/primary time series of interest.Such similarities can be used to evaluate the relationships betweenobjects in the time series, including causality and common afflictions.As used herein, causality refers to the possibility that the similaritybetween time series objects and/or data points indicates that one ormore of the highlighted objects has caused behavior in the subset thathas been evidenced in the baseline time series. Additionally, as usedherein, common afflictions refer to the possibility that the objectsrepresented by the time series data points in the baseline/primary timeseries as well as the candidate time series are exhibiting similarbehavior caused by the same entity or entities.

As further detailed herein, one or more embodiments include implementinga machine learning algorithm that incorporates multiple measures ofsimilarity and/or distance between pairs of time series data points.Such measures include, for example, covariance between time series datapoints, dynamic time warping (DTW) distance between time series datapoints, and shape-based distance (SBD) of time series data points.Covariance between time series data points represents an indicator ofsimilar patterns of rises and falls (in the time series waveforms),incorporating an assumption that the compared time series (waveforms)are aligned. Also, dynamic time warping distance measurements allow forshifts between the time series as determined by a window size inputparameter. Additionally, shape-based distance measurements use anormalized version of a cross-correlation measurement, thereby takinginto consideration the shapes of the time series while comparing them.In connection with at least one embodiment, by combining the output ofcovariance, DTW, and SBD measurements, such an embodiment includesdetermining and/or identifying a set of similar time series that maybenefit from one or more features of each of the individual similaritymeasures.

Because an input time series dataset may include objects represented ondifferent scales, at least one embodiment includes normalizing thedataset. Such an embodiment includes applying z-score normalization toaffect a more comparable evaluation of the time series, as the timeseries are transformed onto a feature space which allows consistentcomparisons.

Once the most similar time series are determined from each of themultiple similarity measures, one or more embodiments includeaggregating the results using an ensemble weighting majority votingtechnique, based on time series profiles across all objects, to selectthose time series whose behavior most closely follows the spikes anddips in the profile of the primary/baseline time series. Using thisinformation, one or more embodiments additionally include identifyingcases of causality and/or common afflictions, as well as remediatingsuch situations and/or issues.

FIG. 2 shows a workflow for determining similarity between time seriesusing machine learning techniques in an illustrative embodiment. By wayof illustration, FIG. 2 depicts machine learning time series similarityalgorithm 205 processing inputs that include primary time series 240, aswell as candidate time series 242-1, 242-2, 242-3, . . . 242-n(hereinafter collectively referred to as candidate time series 242).These inputs are preprocessed by the machine learning time seriessimilarity algorithm 205 via z-score normalization component 230.Subsequently, the preprocessed time series are analyzed via components232 a, 232 b, and 232 c, to determine (for each candidate time series242 relative to the primary time series 240) covariance scores, DTWdistance scores, and SBD distance scores, respectively.

The scores generated by components 232 a, 232 b, and 232 c are providedto component 234, which generates a weighted majority vote, which isthen used to identify the most similar matches among the candidate timesseries 242 to the primary time series 240 (in the FIG. 2 example,candidate time series 242-1 and 242-n are identified as most similar toprimary time series 240).

FIG. 3 shows visual examples of similarity measures utilized in anillustrative embodiment. Specifically, FIG. 3 depicts time series 350 aand 350 b representative of a covariance (also referred to herein as acorrelation) measurement example. As noted above (and depicted in FIG.3), covariance measurements represent indicators of similar patterns ofrises and falls in time series waveforms, incorporating an assumptionthat the compared time series waveforms (e.g., time series 350 a andtime series 350 b in FIG. 3) are aligned. Also, covariance measurementsof normalized data are insensitive to scale differences between timeseries. Accordingly, in at least one embodiment, high covariance scoresrepresent similar patterns of rises and falls across compared timeseries.

Additionally, FIG. 3 depicts time series 352 a and 352 b representativeof a dynamic time warping distance measurement example. As noted above(and depicted in FIG. 3), dynamic time warping distance measurementsallow for shifts in time series data under comparison (e.g., time series352 a and time series 352 b in FIG. 3) as determined by a window sizeinput parameter. Further, FIG. 3 depicts an illustration representativeof a shape-based distance measurement example. As noted above (anddepicted in FIG. 3), shape-based distance measurements use a normalizedversion of a cross-correlation measurement, thereby taking intoconsideration the shapes of the time series being compared (e.g., timeseries 354 a and time series 354 b in FIG. 3). Also, shape-baseddistance measurements are invariant to differences in amplitude andphase.

Referring now to FIG. 4, another illustrative embodiment is shown. Inthis embodiment, pseudocode 400 a-400 b (hereinafter collectivelyreferred to as pseudocode 400) is executed by or under the control of acomputing device and/or processing platform, such as machine learningtime series similarity algorithm 105. For example, pseudocode 400 may beviewed as comprising a portion of a software implementation of at leastpart of machine learning time series similarity algorithm 105 of theFIG. 1 embodiment.

The pseudocode 400 illustrates an example time series similarityalgorithm implemented in one or more embodiments. It is to beappreciated that this particular pseudocode shows just one exampleimplementation of a process for service and workflow definition, andalternative implementations of the process can be used in otherembodiments.

Specifically, pseudocode 400 illustrates an example time seriessimilarity algorithm that uses, as input, data pertaining to a primarytime series and a collection of candidate time series, and generates, asoutput, a list of the most similar time series (from the collection ofcandidates) to the primary time series. As depicted via pseudocode 400,the example time series similarity algorithm includes initiallydetermining if the inputs contain sufficient data. In conjunction withthis determination, the example time series similarity algorithmincludes preprocessing the input data. For example, if there are missingdata points in the input data, the algorithm includes interpolating oneor more of the missing data points. Additionally, the algorithm caninclude removing any candidate time series that exhibits constantactivity or values throughout the given time range. Further, thealgorithm can optionally include filtering stable time series from thecandidate set. It should be appreciated that although the filtering stepis noted as optional, the other steps detailed in pseudocode 400 are notnecessarily required in all embodiments. In such an embodiment, if givena percentile threshold (e.g., 25th percentile), the algorithm includescalculating the average value of the time series data points across allcandidate time series {dataset_mean}. For each candidate time series,the algorithm then includes calculating the difference between itsmaximum and minimum value {max_min_diff}, and sorting the candidate timeseries based on the max_min_diff values. The algorithm subsequentlyincludes filtering out any candidate time series whose max_min_diff islower than the given threshold and whose raw values are always lowerthan the dataset_mean. Also, the preprocessing performed by thealgorithm additionally includes normalizing the data using z-scorenormalization.

Additionally, pseudocode 400 depicts that the algorithm includes scoringthe candidate time series. For each similarity measure(covariance/correlation, dynamic time warping distance, and shape-baseddistance), the algorithm includes calculating the similarity between theprimary time series and all (preprocessed) candidate time series, andranking the candidate time series based on similarity. The algorithmadditionally includes assigning a weight between 0 and 1 to eachcandidate time series, based on its position in the sorted list.Further, for each candidate time series, the algorithm includes summingits weights from each similarity measure to produce a final score,sorting the candidate time series based on their final scores, andreturning a list of the most similar (candidate time series) matches andtheir scores (that is, an output of weighted majority vote).

As further detailed below, the machine learning algorithm(s) describedin connection with one or more embodiments can be implemented inmultiple applications and/or use cases, such as, for example, a noisyneighbors determination for storage objects, a determination of systemresources in contention, and a determination of contributing entities tospecific system behavior.

Additionally, with respect to one or more embodiments, the aspect(s) ofmachine learning incorporated in connection with the time seriessimilarity algorithm(s) includes learning the best value for one or moremodel parameters (such as, as noted herein, the correlation threshold,and the utilization threshold), as well as for any parameters for thesimilarity measures (such as, for example, the window parameter for theDTW distance). Such an embodiment includes learning appropriate valuesto use by evaluating the performance of the algorithm on a set oflabeled data, wherein, for example, a domain expert can specify whichtime series should be included in the final output. Examples of machinelearning techniques that can be utilized in one or more embodimentsinclude time series clustering and k-means clustering. Such embodimentsinclude utilizing such machine learning techniques to determine similarobjects (in particular, time series) as well as distance calculationsbetween pairs of objects.

More specifically, with respect to a determination of system resourcesin contention, when different workloads running on a storage systemutilize the system's resources in a significant manner, the resourcescan begin to experience contention. The contention can manifest, forexample, in the form of a spike in the resources' utilization, which canlead to the performance degradation of workloads which seek to use theover-utilized resources. As in the case of noisy neighbors describedabove, when a workload exhibits a performance degradation, one or moreembodiments can include identifying the resources (e.g., port bandwidth,storage processor utilization, disk input/output operations per second(TOPS), etc.) that are in contention and/or are unable to satisfy theresource requirements of a workload of interest. Such an embodimentincludes comparing the latency time series of the workload of interestto the utilization time series of the storage system's resources. In anexample embodiment, those resources whose spike in utilization mostclosely resembles the spike in latency of the workload in interest canbe identified as the resources in contention.

With respect to implementing the time series similarity algorithmdetailed herein in connection with a resources in contentiondetermination, given the latency time series of a primary workload andthe utilization time series of a set of (candidate) resources, at leastone embodiment includes applying the time series similarity algorithm toproduce an ordered list of most similar-behaving resources, ranked bytheir similarity score. Such an embodiment can include filtering out(candidate) resources whose correlation to the primary time series isless than a given threshold (i.e., a correlation threshold), andcalculating the average utilization of the remaining similar (candidate)resources. Further, such an embodiment includes filtering out resourceswhose average utilization is less than a given threshold (that is, autilization threshold), adding the average utilization value, weightedby a factor (e.g., 6), to each resource's score, and ranking theresources based on their final score.

Additionally, with respect to a determination of contributing entitiesto specific system behavior, multiple key performance indicators (KPIs)are commonly collected at the storage system level, and those KPIs aretypically aggregated from the corresponding metrics at the workloadlevel. For example, a storage system may exhibit a spike in its IOPSprofile, which can be contributed to the spike in TOPS for a set ofworkloads. Accordingly, one or more embodiments include identifying therelevant workloads contributing to the behavioral change in question(e.g., the behavioral change that caused a spike at the storage systemlevel). Such an embodiment includes comparing the TOPS time series ofthe storage system to the TOPS time series of one or more other (e.g.,all other) workloads. In an example embodiment, those workloads whosebehavior is most similar to the behavior at the storage system level canbe identified as the contributing workloads.

With respect to implementing the time series similarity algorithmdetailed herein in connection with a determination of contributingentities to specific system behavior, given the time series for aperformance metric of a primary workload and the time series for thesame performance metric of a set of (candidate) workloads, at least oneembodiment includes applying the time series similarity algorithm toproduce an ordered list of most similar-behaving workloads, ranked bytheir similarity score. Such an embodiment can include filtering out(candidate) workloads whose correlation to the primary time series isless than a given threshold (i.e., a correlation threshold), andcalculating the average value of the performance metric for theremaining similar workloads. Further, such an embodiment includes addingthe logarithm of the average value of the performance metric to eachworkload's score, and ranking the workloads based on their final score.

Further, with respect to a noisy neighbors determination for storageobjects, many workloads may execute simultaneously on a storage system,each competing for resources. At times, divergently behaving (ormisbehaving) workloads may take more than the expected and/or allottedshare of resources, resulting in the performance degradation of one ormore other workloads. Such degradation can manifest, for example, in theform of a spike in the latency (response time) of a particular workloadof interest. When this occurs, at least one embodiment includesidentifying these workloads herein referred to as “noisy neighbors,”that contributed to the performance degradation. Such an embodimentincludes comparing the latency time series of the workload of interestto the TOPS time series of one or more other (e.g., all other)workloads. In an example embodiment, those workloads whose spike in TOPSmost closely resembles the spike in latency of the workload in interestcan be identified as the noisy neighbors.

With respect to implementing the time series similarity algorithmdetailed herein in connection with a noisy neighbors determination,given the latency time series of a primary workload and the TOPS timeseries of a set of (candidate) workloads, at least one embodimentincludes applying the time series similarity algorithm to produce anordered list of most similar-behaving workloads, ranked by theirsimilarity score. Such an embodiment can include filtering out(candidate) workloads whose correlation to the primary time series isless than a given threshold (i.e., a correlation threshold), calculatingthe average TOPS of the remaining similar workloads, adding thelogarithm of the average IOPS to each workload's score, and ranking theworkloads based on their final score.

Storage systems strive to balance the demands of multiple workloadsrunning thereon, each workload with their own unique storage profile.The workloads share the various resources of the storage systems, fromfront-end and back-end adapters, to storage processors and disks. Assuch, care is to be taken to provide adequate resources to all workloadsfor the workloads to be able to achieve their respective service-levelobjectives. If the distribution of resources is not performed properly,workloads and/or the entire storage system may begin to experiencesignificant performance degradations. Accordingly, at least oneembodiment includes automatically identifying divergent and/ormisbehaving workloads and possible causes of such performance impacts.

As detailed herein, such performance degradations can manifest as a risein latency and a drop in the number of IOPS. Accordingly, at least oneembodiment includes implementing a machine learning-based approach toanalyze the input/output (TO) behavior profile of various workloads andthe response time (i.e., latency) profile of a corresponding system or aparticular workload of interest. Such an embodiment includes utilizingmachine learning to correlate deviating behavior of the system orprimary workload with that of one or more misbehaving workloads across arange of time to automatically detect noisy neighbors and/or workloadscontributing to performance degradation within the system.

Because one or more embodiments include analyzing behavior profiles ofdifferent scales (e.g., IO activity versus response time activity), suchan embodiment also includes normalizing the datasets. For example, atleast one embodiment includes applying z-score normalization with anadditional preprocessing step to exclude IO workloads which exhibit lowIO activity. Such preprocessing steps can facilitate identification ofworkloads (e.g., noisy neighbor workloads and/or system performancedegradation contributing workloads) having profiles that not onlyresemble the profile of the affected primary workload or system, butalso represent a non-trivial contribution to the resource contention.

Additionally, one or more embodiments also include utilizing theidentified (contributing) workloads in carrying out one or moreautomated actions (to produce a more effective storage solution, forexample). Such automated actions can include, for example, increasingsystem resources, reducing the workloads on the system, moving workloadsto another system, balancing the load on the system by reschedulingworkloads to be executed at different times, etc.

FIG. 5 shows a workflow for automatic identification of workloadscontributing to system performance degradation using machine learningtechniques in an illustrative embodiment. By way of illustration, FIG. 5depicts a low activity workload filter 555 processing inputs thatinclude primary workload or system time series 540, as well asworkload-1 time series 542-1, workload-2 time series 542-2, workload-3time series 542-3, . . . workload-n time series 542-n (hereinaftercollectively referred to as workload time series 542). The low activityworkload filter 555 excludes, from further analysis and/or processing,any of the workload time series 542 which exhibit a level of 10 activitybelow a predetermined threshold. The output of the low activity workloadfilter 555 (that is, the workload time series exhibiting a level of 10activity at or above the predetermined threshold) is then provided toand processed by the machine learning time series similarity algorithm505 (as detailed herein), which determines (for each workload timeseries 542 (provided to module 505 by the low activity workload filter555) relative to the primary workload or system time series 540)covariance scores, DTW distance scores, and SBD distance scores. Thesescores are then used to generate a weighted majority vote, which is thenused to identify the workload times series among set 542 (in the FIG. 5example, the identified time series include workload time series 542-1and 542-n) most likely to contribute to the performance degradation inquestion (related to the primary workload or system time series 540).

FIG. 6 is a flow diagram of a process for automatic identification ofworkloads contributing to system performance degradation using machinelearning techniques in an illustrative embodiment. It is to beunderstood that this particular process is only an example, andadditional or alternative processes can be carried out in otherembodiments.

In this embodiment, the process includes steps 600 through 608. Step 600includes obtaining, in connection with a system exhibiting performancedegradation, a primary time series and a set of multiple candidate timeseries, wherein the primary time series comprises latency dataattributed to the system or a targeted workload within the system, andwherein the candidate time series comprise input/output activity dataattributed to multiple workloads within the system. In one or moreembodiments, the input/output behavior includes input/output operationsper second. Additionally, at least one embodiment includes normalizingthe primary time series and the multiple candidate time series, whereinnormalizing comprises applying at least one z-score normalizationtechnique to the primary time series and the multiple candidate timeseries. Further, one or more embodiments include filtering any of thecandidate time series exhibiting a level of input/output activity belowa predetermined threshold.

Step 602 includes calculating, using one or more machine learningtechniques, multiple similarity measurements between the primary timeseries and each of the candidate time series in the set. The similaritymeasurements can include a covariance measurement representing anindicator of one or more similar patterns among time series waveforms.Additionally, the similarity measurements can include a dynamic timewarping distance measurement, wherein the dynamic time warping distancemeasurement incorporates one or more shifts between time series asdetermined by a window size input parameter. Further, the similaritymeasurements can include a shape-based distance measurement, wherein theshape-based distance measurement incorporates a normalized version of across-correlation measurement.

Step 604 includes, for each of the multiple similarity measurements,assigning weights to the candidate time series based at least in part onsimilarity to the primary time series relative to the other candidatetime series in the set. Step 606 includes generating, for each of thecandidate time series in the set, a similarity score based at least inpart on the weights assigned to each of the candidate time series acrossthe multiple similarity measurements.

Step 608 includes outputting, based at least in part on the similarityscores, identification of at least one candidate time series from theset for use in one or more automated actions associated with the systemin response to the performance degradation. The one or more automatedactions can include, for example, increasing system resources, reducinga number of workloads on the system, moving one or more of the multipleworkloads from the system to one or more additional systems, and/orrescheduling one or more of the multiple workloads to be executed atdifferent times on the system. Additionally, in at least one embodiment,outputting includes outputting, based at least in part on the similarityscores, identification of the at least one candidate time series fromthe set most similar to the primary time series.

Accordingly, the particular processing operations and otherfunctionality described in conjunction with the flow diagram of FIG. 6are presented by way of illustrative example only, and should not beconstrued as limiting the scope of the disclosure in any way. Forexample, the ordering of the process steps may be varied in otherembodiments, or certain steps may be performed concurrently with oneanother rather than serially.

The above-described illustrative embodiments provide significantadvantages relative to conventional approaches. For example, someembodiments are configured to implement a machine learning algorithmthat incorporates multiple distinct measures of similarity and distancebetween pairs of time series data points corresponding to latency andinput/output activity. These and other embodiments can effectivelyenhance accuracy and scalability for time series analysis correspondingto system performance degradation use cases.

It is to be appreciated that the particular advantages described aboveand elsewhere herein are associated with particular illustrativeembodiments and need not be present in other embodiments. Also, theparticular types of information processing system features andfunctionality as illustrated in the drawings and described above areexemplary only, and numerous other arrangements may be used in otherembodiments.

As mentioned previously, at least portions of the information processingsystem 100 can be implemented using one or more processing platforms. Agiven such processing platform comprises at least one processing devicecomprising a processor coupled to a memory. The processor and memory insome embodiments comprise respective processor and memory elements of avirtual machine or container provided using one or more underlyingphysical machines. The term “processing device” as used herein isintended to be broadly construed so as to encompass a wide variety ofdifferent arrangements of physical processors, memories and other devicecomponents as well as virtual instances of such components. For example,a “processing device” in some embodiments can comprise or be executedacross one or more virtual processors. Processing devices can thereforebe physical or virtual and can be executed across one or more physicalor virtual processors. It should also be noted that a given virtualdevice can be mapped to a portion of a physical one.

Some illustrative embodiments of a processing platform used to implementat least a portion of an information processing system comprises cloudinfrastructure including virtual machines implemented using a hypervisorthat runs on physical infrastructure. The cloud infrastructure furthercomprises sets of applications running on respective ones of the virtualmachines under the control of the hypervisor. It is also possible to usemultiple hypervisors each providing a set of virtual machines using atleast one underlying physical machine. Different sets of virtualmachines provided by one or more hypervisors may be utilized inconfiguring multiple instances of various components of the system.

These and other types of cloud infrastructure can be used to providewhat is also referred to herein as a multi-tenant environment. One ormore system components, or portions thereof, are illustrativelyimplemented for use by tenants of such a multi-tenant environment.

As mentioned previously, cloud infrastructure as disclosed herein caninclude cloud-based systems. Virtual machines provided in such systemscan be used to implement at least portions of a computer system inillustrative embodiments.

In some embodiments, the cloud infrastructure additionally oralternatively comprises a plurality of containers implemented usingcontainer host devices. For example, as detailed herein, a givencontainer of cloud infrastructure illustratively comprises a Dockercontainer or other type of Linux Container (LXC). The containers are runon virtual machines in a multi-tenant environment, although otherarrangements are possible. The containers are utilized to implement avariety of different types of functionality within the system 100. Forexample, containers can be used to implement respective processingdevices providing compute and/or storage services of a cloud-basedsystem. Again, containers may be used in combination with othervirtualization infrastructure such as virtual machines implemented usinga hypervisor.

Illustrative embodiments of processing platforms will now be describedin greater detail with reference to FIGS. 7 and 8. Although described inthe context of system 100, these platforms may also be used to implementat least portions of other information processing systems in otherembodiments.

FIG. 7 shows an example processing platform comprising cloudinfrastructure 700. The cloud infrastructure 700 comprises a combinationof physical and virtual processing resources that are utilized toimplement at least a portion of the information processing system 100.The cloud infrastructure 700 comprises multiple virtual machines (VMs)and/or container sets 702-1, 702-2, . . . 702-L implemented usingvirtualization infrastructure 704. The virtualization infrastructure 704runs on physical infrastructure 705, and illustratively comprises one ormore hypervisors and/or operating system level virtualizationinfrastructure. The operating system level virtualization infrastructureillustratively comprises kernel control groups of a Linux operatingsystem or other type of operating system.

The cloud infrastructure 700 further comprises sets of applications710-1, 710-2, . . . 710-L running on respective ones of theVMs/container sets 702-1, 702-2, . . . 702-L under the control of thevirtualization infrastructure 704. The VMs/container sets 702 compriserespective VMs, respective sets of one or more containers, or respectivesets of one or more containers running in VMs. In some implementationsof the FIG. 7 embodiment, the VMs/container sets 702 comprise respectiveVMs implemented using virtualization infrastructure 704 that comprisesat least one hypervisor.

A hypervisor platform may be used to implement a hypervisor within thevirtualization infrastructure 704, wherein the hypervisor platform hasan associated virtual infrastructure management system. The underlyingphysical machines comprise one or more distributed processing platformsthat include one or more storage systems.

In other implementations of the FIG. 7 embodiment, the VMs/containersets 702 comprise respective containers implemented using virtualizationinfrastructure 704 that provides operating system level virtualizationfunctionality, such as support for Docker containers running on baremetal hosts, or Docker containers running on VMs. The containers areillustratively implemented using respective kernel control groups of theoperating system.

As is apparent from the above, one or more of the processing modules orother components of system 100 may each run on a computer, server,storage device or other processing platform element. A given suchelement is viewed as an example of what is more generally referred toherein as a “processing device.” The cloud infrastructure 700 shown inFIG. 7 may represent at least a portion of one processing platform.Another example of such a processing platform is processing platform 800shown in FIG. 8.

The processing platform 800 in this embodiment comprises a portion ofsystem 100 and includes a plurality of processing devices, denoted802-1, 802-2, 802-3, . . . 802-K, which communicate with one anotherover a network 804.

The network 804 comprises any type of network, including by way ofexample a global computer network such as the Internet, a WAN, a LAN, asatellite network, a telephone or cable network, a cellular network, awireless network such as a Wi-Fi or WiMAX network, or various portionsor combinations of these and other types of networks.

The processing device 802-1 in the processing platform 800 comprises aprocessor 810 coupled to a memory 812.

The processor 810 comprises a microprocessor, a microcontroller, anapplication-specific integrated circuit (ASIC), a field-programmablegate array (FPGA) or other type of processing circuitry, as well asportions or combinations of such circuitry elements.

The memory 812 comprises random access memory (RAM), read-only memory(ROM) or other types of memory, in any combination. The memory 812 andother memories disclosed herein should be viewed as illustrativeexamples of what are more generally referred to as “processor-readablestorage media” storing executable program code of one or more softwareprograms.

Articles of manufacture comprising such processor-readable storage mediaare considered illustrative embodiments. A given such article ofmanufacture comprises, for example, a storage array, a storage disk oran integrated circuit containing RAM, ROM or other electronic memory, orany of a wide variety of other types of computer program products. Theterm “article of manufacture” as used herein should be understood toexclude transitory, propagating signals. Numerous other types ofcomputer program products comprising processor-readable storage mediacan be used.

Also included in the processing device 802-1 is network interfacecircuitry 814, which is used to interface the processing device with thenetwork 804 and other system components, and may comprise conventionaltransceivers.

The other processing devices 802 of the processing platform 800 areassumed to be configured in a manner similar to that shown forprocessing device 802-1 in the figure.

Again, the particular processing platform 800 shown in the figure ispresented by way of example only, and system 100 may include additionalor alternative processing platforms, as well as numerous distinctprocessing platforms in any combination, with each such platformcomprising one or more computers, servers, storage devices or otherprocessing devices.

For example, other processing platforms used to implement illustrativeembodiments can comprise different types of virtualizationinfrastructure, in place of or in addition to virtualizationinfrastructure comprising virtual machines. Such virtualizationinfrastructure illustratively includes container-based virtualizationinfrastructure configured to provide Docker containers or other types ofLXCs.

As another example, portions of a given processing platform in someembodiments can comprise converged infrastructure.

It should therefore be understood that in other embodiments differentarrangements of additional or alternative elements may be used. At leasta subset of these elements may be collectively implemented on a commonprocessing platform, or each such element may be implemented on aseparate processing platform.

Also, numerous other arrangements of computers, servers, storageproducts or devices, or other components are possible in the informationprocessing system 100. Such components can communicate with otherelements of the information processing system 100 over any type ofnetwork or other communication media.

For example, particular types of storage products that can be used inimplementing a given storage system of a distributed processing systemin an illustrative embodiment include all-flash and hybrid flash storagearrays, scale-out all-flash storage arrays, scale-out NAS clusters, orother types of storage arrays. Combinations of multiple ones of theseand other storage products can also be used in implementing a givenstorage system in an illustrative embodiment.

It should again be emphasized that the above-described embodiments arepresented for purposes of illustration only. Many variations and otheralternative embodiments may be used. Also, the particular configurationsof system and device elements and associated processing operationsillustratively shown in the drawings can be varied in other embodiments.Thus, for example, the particular types of information processingsystems, host devices and storage systems deployed in a given embodimentand their respective configurations may be varied. Moreover, the variousassumptions made above in the course of describing the illustrativeembodiments should also be viewed as exemplary rather than asrequirements or limitations of the disclosure. Numerous otheralternative embodiments within the scope of the appended claims will bereadily apparent to those skilled in the art.

What is claimed is:
 1. A computer-implemented method comprising:obtaining, in connection with a storage system exhibiting performancedegradation, a primary time series and a set of multiple candidate timeseries, wherein the primary time series comprises latency dataattributed to the storage system or a targeted workload within thestorage system, and wherein the candidate time series compriseinput/output activity data attributed to multiple workloads within thestorage system; calculating, using one or more machine learningtechniques, multiple similarity measurements between the primary timeseries and each of the candidate time series in the set, wherein themultiple similarity measurements comprise a covariance measurementrepresenting an indicator of one or more similar pattern alignmentsamong time series waveforms, a dynamic time warping distancemeasurement, wherein the dynamic time warping distance measurementincorporates one or more shifts between time series as determined by awindow size input parameter, and a shape-based distance measurement,wherein the shape-based distance measurement incorporates a normalizedversion of a cross-correlation measurement and processes and comparesshapes of time series waveforms; for each of the multiple similaritymeasurements, assigning weights to the candidate time series based atleast in part on similarity to the primary time series relative to theother candidate time series in the set; generating, for each of thecandidate time series in the set, a similarity score based at least inpart on the weights assigned to each of the candidate time series acrossthe multiple similarity measurements; and outputting, based at least inpart on the similarity scores, identification of at least one candidatetime series from the set for use in one or more automated actionsassociated with the storage system in response to the performancedegradation; wherein the method is performed by at least one processingdevice comprising a processor coupled to a memory.
 2. Thecomputer-implemented method of claim 1, further comprising: normalizingthe primary time series and the multiple candidate time series.
 3. Thecomputer-implemented method of claim 2, wherein normalizing comprisesapplying at least one z-score normalization technique to the primarytime series and the multiple candidate time series.
 4. Thecomputer-implemented method of claim 1, further comprising: filteringany of the candidate time series exhibiting a level of input/outputactivity below a predetermined threshold.
 5. The computer-implementedmethod of claim 1, wherein the input/output behavior comprisesinput/output operations per second.
 6. The computer-implemented methodof claim 1, wherein the one or more automated actions compriseincreasing storage system resources.
 7. The computer-implemented methodof claim 1, wherein the one or more automated actions comprise reducinga number of workloads on the storage system.
 8. The computer-implementedmethod of claim 1, wherein the one or more automated actions comprisemoving one or more of the multiple workloads from the storage system toone or more additional storage systems.
 9. The computer-implementedmethod of claim 1, wherein the one or more automated actions compriserescheduling one or more of the multiple workloads to be executed atdifferent times on the storage system.
 10. A non-transitoryprocessor-readable storage medium having stored therein program code ofone or more software programs, wherein the program code when executed byat least one processing device causes the at least one processingdevice: to obtain, in connection with a storage system exhibitingperformance degradation, a primary time series and a set of multiplecandidate time series, wherein the primary time series comprises latencydata attributed to the storage system or a targeted workload within thestorage system, and wherein the candidate time series compriseinput/output activity data attributed to multiple workloads within thestorage system; to calculate, using one or more machine learningtechniques, multiple similarity measurements between the primary timeseries and each of the candidate time series in the set, wherein themultiple similarity measurements comprise a covariance measurementrepresenting an indicator of one or more similar pattern alignmentsamong time series waveforms, a dynamic time warping distancemeasurement, wherein the dynamic time warping distance measurementincorporates one or more shifts between time series as determined by awindow size input parameter, and a shape-based distance measurement,wherein the shape-based distance measurement incorporates a normalizedversion of a cross-correlation measurement and processes and comparesshapes of time series waveforms; to assign, for each of the multiplesimilarity measurements, weights to the candidate time series based atleast in part on similarity to the primary time series relative to theother candidate time series in the set; to generate, for each of thecandidate time series in the set, a similarity score based at least inpart on the weights assigned to each of the candidate time series acrossthe multiple similarity measurements; and to output, based at least inpart on the similarity scores, identification of at least one candidatetime series from the set for use in one or more automated actionsassociated with the storage system in response to the performancedegradation.
 11. The non-transitory processor-readable storage medium ofclaim 10, wherein the input/output behavior comprises input/outputoperations per second.
 12. The non-transitory processor-readable storagemedium of claim 10, wherein the one or more automated actions compriseat least one of increasing storage system resources, reducing a numberof workloads on the storage system, moving one or more of the multipleworkloads from the storage system to one or more additional storagesystems, and rescheduling one or more of the multiple workloads to beexecuted at different times on the storage system.
 13. Thenon-transitory processor-readable storage medium of claim 10, whereinthe program code when executed by the at least one processing devicecauses the at least one processing device: to normalize the primary timeseries and the multiple candidate time series.
 14. The non-transitoryprocessor-readable storage medium of claim 13, wherein normalizingcomprises applying at least one z-score normalization technique to theprimary time series and the multiple candidate time series.
 15. Anapparatus comprising: at least one processing device comprising aprocessor coupled to a memory; the at least one processing device beingconfigured: to obtain, in connection with a storage system exhibitingperformance degradation, a primary time series and a set of multiplecandidate time series, wherein the primary time series comprises latencydata attributed to the storage system or a targeted workload within thestorage system, and wherein the candidate time series compriseinput/output activity data attributed to multiple workloads within thestorage system; to calculate, using one or more machine learningtechniques, multiple similarity measurements between the primary timeseries and each of the candidate time series in the set, wherein themultiple similarity measurements comprise a covariance measurementrepresenting an indicator of one or more similar pattern alignmentsamong time series waveforms, a dynamic time warping distancemeasurement, wherein the dynamic time warping distance measurementincorporates one or more shifts between time series as determined by awindow size input parameter, and a shape-based distance measurement,wherein the shape-based distance measurement incorporates a normalizedversion of a cross-correlation measurement and processes and comparesshapes of time series waveforms; to assign, for each of the multiplesimilarity measurements, weights to the candidate time series based atleast in part on similarity to the primary time series relative to theother candidate time series in the set; to generate, for each of thecandidate time series in the set, a similarity score based at least inpart on the weights assigned to each of the candidate time series acrossthe multiple similarity measurements; and to output, based at least inpart on the similarity scores, identification of at least one candidatetime series from the set for use in one or more automated actionsassociated with the storage system in response to the performancedegradation.
 16. The apparatus of claim 15, wherein the input/outputbehavior comprises input/output operations per second.
 17. The apparatusof claim 15, wherein the one or more automated actions comprise at leastone of increasing storage system resources, reducing a number ofworkloads on the storage system, moving one or more of the multipleworkloads from the storage system to one or more additional storagesystems, and rescheduling one or more of the multiple workloads to beexecuted at different times on the storage system.
 18. The apparatus ofclaim 15, wherein the at least one processing device is furtherconfigured: to normalize the primary time series and the multiplecandidate time series.
 19. The apparatus of claim 18, whereinnormalizing comprises applying at least one z-score normalizationtechnique to the primary time series and the multiple candidate timeseries.
 20. The apparatus of claim 15, wherein the at least oneprocessing device is further configured: to filter any of the candidatetime series exhibiting a level of input/output activity below apredetermined threshold.